Control device and control method for robot

ABSTRACT

A controlling apparatus for a robot of the type formed by a plurality of joint actuators and operating in accordance with a behavioral schedule comprises a behavior scheduling unit for setting a robot&#39;s behavioral schedule, an operation controller for implementing an operational pattern corresponding to the behavioral schedule as determined by the behavior scheduling unit by driving each joint actuator, a detector for detecting the state of operation implementation by the operation controller and a recording unit for recording a log including the behavioral schedule by the behavior scheduling unit and the state of operation implementation by the detector. A user issuing a command for the robot is authenticated and the contents of the command supplied from the user are recorded in combination with the behavior taken by the robot responsive to the command and the time point of implementation of the behavior.

TECHNICAL FIELD

This invention relates to a robot operating in accordance with abehavioral schedule and, in particular, it relates to a behaviorschedule setting type robot which sets a behavioral schedule byinteraction with a user based on inputting of the speech or an image orautonomously without recourse to user inputs. More specifically, itrelates to a behavior schedule setting type robot which is able to checkfor a cause of abnormalities, malfunctions or troubles if such shouldoccur during the period of “dialogue driving” or “autonomous driving”.

BACKGROUND ART

A mechanical apparatus for performing movements simulating the movementof the human being using electrical or magnetic operation is termed a“robot”. The etymology of the term robot is said to be “ROBOTA” (slavemachine) of the Slavic language. The robots started to be usedextensively towards the end of sixtieth. Most of the robots used wereindustrial robots, such as manipulators or transporting robots, aimed atautomating or performing unmanned operations in plant operations.

The standstill type robot, installed and used at a fixed place, such asarmed robots, are in operation only in a stationary or local workingspace such as for assembling or sorting of component parts. On the otherhand, the mobile robots are not limited as to working space and aremovable on a preset or undefined path in an unrestricted fashion toperform operations to take the place of human operators or to offervariegated services to take the place of the human being, dogs or otherliving organisms. The legged mobile robots, while being unstable anddifficult to control as to its orientation or walking, as compared tocrawler or tire type robots, are superior in climbing up and down aladder or a staircase, in riding over obstacles or walking or runningflexibly on leveled or unleveled terrain.

In recent years, researches and development in legged mobile robots,including pet type robots, simulating the bodily mechanism or movementsof animals, such as quadruples, e.g., dogs or cats, or so-calledhumanoid robots, simulating the bodily mechanism or movements of animalserected and walking on feet, such as human being, are progressing, andexpectations may be made of practical utilization of these robot types.

One usage of the legged mobile robot is taking the place of humanoperators in a variety of difficult operations in industrial andproductive activities, such as, for example, taking the place of thehuman operators in maintenance operations in nuclear power plants,thermal power generation plants or in petrochemical plants, transportand assembly operations in manufacturing plants, cleaning in high-risebuildings or rescue on the sites of conflagration.

Another usage of the legged mobile robot is the living coherent typeusage, rather than the aforementioned operation substitution type usage,that is the usage for “co-living” with the human being. This type of therobot emulates the behavioral mechanism or the feeling expression of theanimal of a higher intellect, walking on legs, such as human being ordogs kept as pets. This type of the robot is also required not only toimplement previously input behavior patterns faithfully, but also torealize expressions vivid actions responsive to the speech or demeanorof the user, such as praising, scolding, or hitting.

The conventional toy machines are fixed as to the relation between auser operation and a responsive operation, such that the movement of atoy cannot be changed to the user's liking. The result is that the useris tired of the toy, doing nothing but repeating the same operations,sooner or later.

Conversely, an intelligent robot implements intellectual operations, inaccordance with a chronological model of the operation generation, inaddition to doing autonomous thinking control and autonomous control ofactions. Moreover, the robot, equipped with an image inputting device ora speech input/output device and doing image or speech processing, isable to perform the realistic communication with the human being at ahigher intellectual level. The intelligent robot is responsive todetection of a stimulus from outside, such as user actuation, to changethe chronological model. That is, by affording the “learning effect”such a behavior pattern may be provided which is adapted to the taste ofthe user and hence is not tiresome to the user. Moreover, the user isable to enjoy a sort of the inculcation simulation with a game feeling.

In the case of a robot dedicated to a special industrial usage, it issufficient if the user or the operator inputs a command forimplementation which can be interpreted subsequent unequivocally. Thisis analogous to the faithful responsive operation of an informationprocessing equipment to a console input of a unique command, such asfile copying or deletion, or file opening.

On the other hand, the co-living or entertainment type robot is able toimplement “dialogue driving” in which the robot operates on the basisnot only of a definitive or unequivocal command from the user but alsoon more abstract input data, such as speech or image, and “autonomousdriving” in which the user operates in accordance with the behaviorschedule set by no other than the robot, without recourse to the commandfrom or dialog with the user, that is independently of the user.

However, the higher the degree of freedom in actions or functionality ofthe robot, such as by causing the robot to interpret abstract userinputs or thinking control proper to the robot, the more difficult itbecomes to search into the cause of malfunctions or troubles if such areproduced.

In routine machines or apparatus, the response of the apparatus to aninput command is subsequent in a one-for-one correspondence, so that itcan be determined readily which command input or implemented has causedan abnormality in the apparatus.

Conversely, with the aforementioned “dialogue driving” or “autonomousdriving” type robot, in which there is permitted a certain latitude inthe interpretation on the part of the robot of the user input orexternal events, it is difficult to locate the cause of the abnormalityor malfunctions. Additionally, since the legged mobile robot can walk inan arbitrary working space without following a fixed route, the user isunable to monitor the robot at all times. Therefore, if, as the robot isput outside the monitoring range by the user, there has occurred amalfunction, or the robot has become involved in troubles or accidents,the user will find it extremely difficult to search what has happened tothe robot.

DISCLOSURE OF THE INVENTION

It is an object of the present invention to provide a behaviorscheduling type robot that is able to set a behavioral schedule by adialog with a user based on an input such as speech or image, or that isable to autonomously set a behavioral schedule without recourse to auser input.

It is another object of the present invention to provide a behaviorscheduling type robot that is able to search into a cause ofabnormalities, malfunctions or troubles in which the robot is involvedduring “dialog driving” or “autonomous driving”.

For accomplishing the above object, the present invention provides acontrolling apparatus for a robot of the type formed by a plurality ofjoint actuators and operating in accordance with a behavioral schedule,including a behavior scheduling unit for setting a robot's behavioralschedule, an operation controller for implementing an operationalpattern corresponding to the behavioral schedule as determined by thebehavior scheduling unit by driving each joint actuator, a detector fordetecting the state of operation implementation by the operationcontroller, and a recording unit for recording a log including thebehavioral schedule by the behavior scheduling unit and the state ofoperation implementation by the detector.

The controlling apparatus for a robot according to the present inventionfurther includes a user input unit for receiving a command or data fromthe user, and a dialog management unit for supervising the dialog withthe user based on the user input command or data from the user inputunit. The behavior scheduling unit may set a behavioral schedule inaccordance with the contents of the dialog in the dialog managementunit, while the recording unit may take a log of the dialog contents.

The controlling apparatus for a robot according to according to thepresent invention may further include a self-diagnosis unit fordiagnosing each part of the robot. The recording unit may take the logof the results of the self-diagnosis.

The controlling apparatus for a robot according to the present inventionmay further include a user authentication unit for authenticating a userlying in the vicinity of the robot. The recording unit may take a log ofthe results of user authentication.

The behavior scheduling unit may be operable in accordance with a firstoperating system of setting a behavioral schedule based on a dialog withthe user or in accordance with a second operating system of setting abehavioral schedule based on a feeling model as determined responsive toexternal changes. In such case, the recording unit taking a log of anoperating system in the behavior scheduling unit.

The robot according to the present invention, is able to search anarbitrary working space routlessly, as a legged mobile robot does. Thisworking space can be co-owned with the living space of the human being.This robot is able to set a behavioral schedule by dialog with the userbased on audio or image input or to set a behavioral scheduleautonomously without recourse to the user input.

The robot of the present invention is able to authenticate a user whoissues a command to the robot, or to extract the face, voice or otherbiological features of the user who cannot be authenticated. Moreover,the contents of the command issued by the user, the behavior taken bythe robot responsive to the command and the time of implementation ofthe behavior can be combined together and recorded as a sort of the log.The inner state of the robot or the sensor input information can also berecorded together.

The contents of the log can be analyzed later to investigate into thecause of abnormalities, malfunctions or troubles in which the robot isinvolved.

The history of the robot's actions, such as “what is the action therobot did, when and with whom such action was done”, can be searchedeven during the time other than the time of the abnormalities,malfunctions or troubles. Alternatively, the history of the acts of therobot can be supervised as the empirical information so that theinformation on “what is the action the robot did, when and with whomsuch action was done” can be extracted from the dialog between the userand the robot. Such robot is highly entertainment-oriented. In addition,the user can enjoy a sort of an inculcation simulation with the sense ofa game.

Other objects, features and advantages of the present invention willbecome more apparent from reading the embodiments of the presentinvention as shown in the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a legged mobile robot of the presentinvention when seen from the front side.

FIG. 2 is a perspective view of the legged mobile robot of the presentinvention when seen from the back side.

FIG. 3 schematically shows a freedom degree forming model provided inthe legged mobile robot of the present invention.

FIG. 4 schematically shows a control system of the legged mobile robotof the present invention.

FIG. 5 is a block diagram showing a typical functional block structurefor recording the behavioral recording or empirical information in thelegged mobile robot of the present invention.

FIG. 6 is a block diagram showing another typical functional blockstructure for recording the behavioral recording or empiricalinformation in the legged mobile robot of the present invention.

FIG. 7 shows a typical data format of an behavior log.

FIG. 8 shows an operating state that can be taken by the robot of thepresent invention.

FIG. 9 schematically shows an operational sequence of a robot duringactive driving.

BEST MODE FOR CARRYING OUT THE INVENTION

Referring to the drawings, a robot employing a control apparatusaccording to the present invention and a control method therefor areexplained in detail.

FIGS. 1 and 2 show a legged mobile robot 100 of the present invention,in the erect state, with the robot being seen from the front and backsides, respectively. This legged mobile robot 100 is of the type called“humanoid” and is able to set a behavior schedule through a dialog witha user based on- a speech or video input, or autonomously, withoutrecourse to the user inputs, that is independently of the user, as willbe explained subsequently. As shown in FIGS. 1 and 2, the legged mobilerobot 100 includes two lower limbs 101R, 101L, responsible for movementon legs, a body trunk 102, left and right upper limbs 103R, 103L and ahead 104.

The left and right lower limbs 101R, 101L are made up of thighs 105R,105L, knee joints 106R, 106L, shins 107R, 107L, ankles 108R, 108L andfoot flats 109R, 109L. The left and right lower limbs 101R, 101L areconnected by hip joints 110R, 110L at approximately the lowermost pointsof the body trunk 102. The left and right upper limbs 103R, 103L aremade up of upper arms 111R, 111L, knee joints 112R, 112L, and forearms113R, 113L, and are connected by shoulder joints 114R, 114L at left andright side edges of the body trunk 102. The head 104 is connected by aneck joint 155 to approximately the uppermost center point of the bodytrunk 102.

Within the body trunk unit is mounted a control unit not shown in FIG. 1or FIG. 2. This controller is a casing carrying a controller forming adriving control for each joint actuator forming the legged mobile robot100 and a main controller for processing an exterior input fromrespective sensors as later explained and peripherals such as powersource circuitry. The control unit may include a communication interfaceor communication device for remote control.

FIG. 3 schematically shows the structure of the degree of joint freedomowned by the legged mobile robot 100 of the present invention. As shown,the legged mobile robot 100 is made up of an upper body portionincluding two arm and a head 1, a lower limb comprised of two legs forimplementing the movement actions, and a body trunk portioninterconnecting the upper limb and the lower limb.

The neck joint, supporting the head 1, has three degrees of freedom,namely a neck joint yaw axis 2, a neck joint pitch axis 3 and a neckjoint roll axis 4.

Each arm is made up of a shoulder joint pitch axis 8, a shoulder jointroll axis 9, an upper arm yaw axis 10, a hinge joint pitch axis 11, aforearm yaw axis 12, a wrist joint pitch axis 13, a wrist joint rollaxis 14 and a hand 15. The hand 15 is, in actuality, a multi-jointmulti-freedom structure including plural fingers. It is however assumedthat the movement of the hand 15 itself is assumed to have a zero degreeof freedom because it contributes to or influences the orientationstability control or walking movement control of the robot 100 only to alesser extent. Therefore, the left and right arms are assumed to haveseven degrees of freedom.

The body trunk portion has three degrees of freedom, namely a body axispitch axis 5, a body trunk roll axis 6 and a body trunk yaw axis 7.

Left and right legs, forming the lower limb, are each made up of a hipjoint yaw axis 16, a hip joint pitch axis 17, a hip joint roll axis 18,a knee joint pitch axis 19, an ankle joint pitch axis 20, an ankle jointroll axis 21 and a foot (foot sole of foot flat). The point ofintersection of the hip joint pitch axis 17 and the hip joint roll axis18 defines a hip joint position of the robot 100 of the presentembodiment. Although the foot (foot sole) 22 of the human body is, ineffect, a structure including the multi-joint multi-freedom foot sole,the foot sole of the legged mobile robot 100 of the present invention isassumed to be of zero degree of freedom. Thus, the left and right feetare made up of six degrees of freedom.

In sum, the legged mobile robot 100, in its entirety, has 3+7×2+3+6×2=32degrees of freedom. However, the legged mobile robot 100 is notnecessarily limited to the 32 degrees of freedom. The degrees offreedom, that is the number of joints, can, of course, be optionallyincreased or decreased depending on designing and fabrication constraintor design parameter requirements.

In actuality, the above-described respective degrees of freedom of thelegged mobile robot 100 are realized as active movements by jointactuators. From a variety of requirements for eliminating any excessbulging portions in the appearance of the overall device to simulate theshape of the body of the human being, and for exercising orientationcontrol on an unstable structure for realizing walking on two feet, thejoint actuators are desirably small-sized and lightweight.

According to the present invention, a small-sized AC servo actuator isused, which is of the type directly coupled to a gearing and which has aone-chip servo control system enclosed in a motor unit. Meanwhile, thesmall-sized AC servo actuator, applicable to a legged robot, isdisclosed in, for example, the specification of the JP PatentApplication H-11-33386 transferred to and filed in the name of thepresent Assignee.

FIG. 4 schematically shows a control system configuration of the leggedmobile robot 100 according to the present invention. As shown therein,the control system is made up of a thinking control module 200,dynamically reacting to a user input to take charge of the sentimentjudgment or feeling expression, and a movement control module 300controlling the whole body concerted movement of the robot 100 such asdriving of the joint actuator.

The thinking control module 200 is an independent information processingapparatus, comprised of a CPU (central processing unit) 211 forexecuting calculation processing concerning sentiment judgment orfeeling expression, a RAM (random access memory) 212, a ROM (read-onlymemory) 213 and hard disc drive such as an exterior storage devices 214.The processing may be self-complete in the module 200. It is possible tostore the walking pattern or other operational patterns calculatedoff-line, such as walking patterns, within the exterior storage devices214.

To the thinking control module 200, there are connected, through a businterface 201, a variety of units, including a picture input device 251,such as a CCD (charge-coupled device) camera provided in the head part201, a speech input device 252, such as a microphone, a speech outputdevice 253, such as a loudspeaker, or a communication interface 254 foreffecting data exchange with a system outside the robot 100 throughe.g., LAN (local area network), not shown.

The thinking control module 200 decides the current feeling or will ofthe legged mobile robot 100, in accordance with stimuli from the outsideworld or changes in the environment, such as pictures or visual datainput from the picture input device 251 or speech or acoustic data inputfrom the speech input device 252. The thinking control module 200 alsoissues a command to the movement control module 300 to implement thebehavior or movement corresponding to the decision of will, that ismovements of the four limbs 322R, 322L, 331R and 331L.

The movement control module 300 is comprised of a CPU (centralprocessing unit) 311 for controlling the whole body concerted movementsof the robot 100, a RAM (random access memory) 312, a ROM (read-onlymemory) 313 and an exterior storage devices 314 such as a hard discdrive. The processing may be self-complete within the module 300. Theexterior storage devices 314 is able to store e.g., the behavioralpattern or the “walking capacity” employing the four limbs. The “walkingcapacity” is a technical term used in the related art to denote“chronological changes of the joint angle”.

To the movement control module 300, there are connected, through a businterface 301, a variety of devices, such as a joint actuator 321 forrealizing respective degrees of freedom of the joints distributed in thewhole body of the robot 100 (see FIG. 9), an orientation sensor 351 formeasuring the orientation and the tilt of the body trunk 202, roadsurface contact sensors 361 and relative movement measurement sensors362 provided on left and right feet and power source control devices forsupervising the power source such as battery.

The movement control module 300 controls the whole body concertedmovement by the joint actuators 321, 335 in order to implement thebehavior commanded by the thinking control module 200. That is, the CPU311 fetches the behavioral pattern corresponding to the behavior ascommanded by the thinking control module 200 from the exterior storagedevice 314 or internally generates a movement pattern. The CPU 311 alsosets the foot movement, ZMP (zero moment point) trajectory, body trunkmovement, upper limb movement, and the horizontal waist position andheight, in accordance with the specified behavioral pattern, whilesending command values specifying the movements conforming to the as-setcontents to the respective joint actuators.

The CPU 311 is able to adaptively control the whole body concertedmovement of the legged mobile robot 100 by detecting the orientation ortilt of the body trunk part of the robot 100 by output signals of theorientation sensor 351 and by detecting whether the respective mobilelegs are in the free state or in the erected state by output signals ofthe road surface contact sensors 361 of the left and right legs.

Moreover, the movement control module 300 is adapted for returning towhich extent the behavior has been implemented up to the will determinedby the thinking control module 200, that is the processing state, to thethinking control module 200. The thinking control module 200 and themovement control module 300 are constructed on a common platform and areinterconnected over bus interfaces 201, 301.

The legged mobile robot 100 is adapted for recording the contents of acommand issued by the user in combination with the behavior taken by therobot and the time of taking the behavior. Thus, the user is able toanalyze the recorded contents later to search the abnormalities ormalfunctions that occurred in the robot and the causes of such troubles.Additionally, the behavioral history of the robot, such as where, withwhom and what the robot did during the time other than the time of theabnormalities, malfunctions and troubles, can be investigated. In thefollowing, the recording management processing of the behavioral historyor empirical information of the robot in the present invention isexplained.

FIG. 5 shows a typical functional block for implementation of thebehavioral recording or the empirical information in the legged mobilerobot 100. The robot 100 shown in FIG. 5 is able to perform operationsof the “dialog driving” type in which the robot sets a behavioralschedule such as to follow the dialog with the user based on the speechor image input.

For user authentication, the legged mobile robot 100 uses at least oneof a fingerprint sensor 111 or a visual sensor 113.

As the visual sensor 113, the image input device 251, such as a CCD(charge coupled device) loaded on the head part may be used together. Inthis case, one or more face images of the registered user, images ofother body parts of the user, or the feature information extracted fromthe images, are stored in advance. An authentication processor 114processes e.g., the face images input from the visual sensor 113 andcompares them to previously registered face images or the featureinformation for collation to effect authentication processing as towhether or not the user is an authorized user.

Although not shown in FIGS. 1 to 4, a fingerprint sensor 111 includes afingerprint readout head on a site on which the user can easily touchwith a finger, such as head part 104 or a shoulder part. One of morefingerprints of the registered user are previously stored as theauthentication information. An authentication processor 112 performsimage processing and feature extraction processing on the fingerprintinformation input from the fingerprint sensor 111, and compares theinput fingerprint information to the pre-registered fingerprint forcollation to effect authentication processing to determine whether ornot the user is an authorized user. Meanwhile, the Japanese Laying-OpenPatent Publication H-3-18980 discloses a fingerprint collation deviceprovided with a reference image memory holding image data of thereference fingerprint and an image input device inputting the image ofthe fingerprint to be authenticated and which collates the fingerprintas the two images are superposed together and checked for the degree ofcoincidence.

The fingerprint sensor 111 and the authentication processor 112 are ableto perform more accurate authentication processing for the user whodefinitely requested fingerprint collation. On the other hand, thevisual sensor 113 and the authentication processor 114 are able toperform authentication processing, in the absence of the request on thepart of the user, with the user being not aware of it, as the user istracked and imaged with a camera. The authentication processors 112, 114are also able to transfer the results of authentication of the user toan information recording management unit 125.

In order for the user to input commands or data, the legged mobile robot100 of the present invention furnishes four means, namely a speechrecognition unit 115, a communication control unit 116, an imagerecognition unit 117 and a commander 118.

The speech recognition unit 115 is made up of a speech input unit 252,such as a microphone, a CPU 211 capable of recognizing and processingthe speech, or other calculation processing circuit. The user is able toinput commands, which are in natural language form or are abstruse, suchas “run”, “carry οΔ” or “hurry up” by speech. The speech recognitionunit 115 recognizes the user's speech. A command interpreting unit 119comprehends and analyzes the command to the robot 100 based on theresult of speech recognition.

The communication control unit 116 is made up of a communicationinterface 254, and a CPU 211 for processing the command or dataexchanged through the communication interface 254 or other calculationprocessing circuits. The communication interface 254 is interconnectedto an information terminal, such as a personal computer, through aBluetooth or other wireless data communication network. The user is ableto input the commands of the form that can be uniquely interpreted by anautomated machine, such as a robot, or the commands of more abstruseform, at a console on such information terminal for transfer to therobot 100. The user command received by the communication control unit116 is interpreted by the command interpreting unit 119 and thencetransferred to a dialog management unit 124.

The image recognition unit 117 is made up of the image input device 251,such as a CCD camera, and a CPU 211, capable of image recognitionprocessing or the like calculating circuit. The user is able to input acommand such as gesture or hand movements. The image recognition unit117 recognizes an images while the command interpreting unit 119comprehends and analyzes the gesture or hand movements as a commandbased on the recognized result. The command may also be input nor by thegesture but by inputting the visible information, such as cyber-codes,uniquely assigned to the commands, as an image to input a command.

Although not shown in particular in FIGS. 1 to 4, the commander 118 isconstructed as a key/button type user input device, loaded on the backor the belly of the legged mobile robot 100, or as a remote controller,not shown. The user can enter a command of a form that can be uniquelyinterpreted by a robot, or a command of more abstruse form, on thecommander 118. The user command, input on the commander 118, isinterpreted by the command interpreting unit 119 and thence sent to thedialog management unit 124.

The dialog management unit 124 chronologically manages the commands ofvariable forms, received through the command interpreting unit 119, tocomprehend the context of the dialog with the user, as well as tomaintain the dialog context. In the present embodiment, the dialogmanagement unit 124 is able to transfer the contents of the dialog withthe user to the information recording management unit 125. The dialogmanagement unit 124 is also able to generate a reply to the user inputin a speech synthesis unit 120 and subsequently outputs the reply tooutside via a loudspeaker or outputs the reply at a console at e.g., anexternal computer system, such as a user terminal, through acommunication controller 121 or as an image through a GUI.

The speech synthesis unit 120 is made up of a speech outputting unit253, such as a loudspeaker, a CPU 211 that is able to synthesize thespeech, or the like calculation circuit. The communication controller121 is made up of a communication interface 254, a CPU 211, forprocessing commands or data exchanged over the communication interface254, and other calculating circuits.

An orientation sensor 130, a temperature sensor 131, a joint anglesensor 132, a contact sensor 133, a force sensor 134, a power sourcemanagement unit 135 and a communication controlling detection unit 136are functional modules for detecting the outside field or environment ofthe robot 100, changes in such outside field or environment, and thestate of operation implementation.

The orientation sensor 130 is equivalent to a reference numeral 354 inFIG. 4, and is able to detect the orientation of the robot 100 withrespect to the robot 100. The joint angle sensor 132 is equivalent tothe rotary encoder mounted in each joint actuator (see FIG. 3). Thecontact sensor 133 is equivalent to ground touching confirming sensors352, mounted on the foot parts. Although not shown in FIGS. 1 to 4, theforce sensor 134 is mounted in each part of the whole body of the robot100 for detecting the possible conflict against the user or obstacles ofthe outer field.

The power source management unit 135 is equivalent to the referencenumeral 354 in FIG. 4, and is designed to monitor the power sourcevoltage, current or the power source capacity of the battery as a mainpower source of the robot 100. The communication controlling detectionunit 136 is made up of a communication interface 254 and a CPU 211 orthe like calculation circuit for processing commands or data exchangedover the communication interface 254, and detects changes in the outerfield or the environment by the communication commands or communicationdata.

The environmental factors, prescribed by these sensors, are sent to abehavior scheduling unit 127 and to a self-diagnosis unit 129.

The behavior scheduling unit 127 sets a behavioral schedule for therobot 100, in accordance with the context of the dialog in the dialogmanagement unit 124 or the outer field or the environment or the robot100, prescribed by a sensor functional module 130, and commands anoperation controller 128 to implement the behavior. According to thepresent invention, the behavior scheduling unit 127 is able to transferthe contents of the behavioral schedule to the information recordingmanagement unit 125.

The operation controller 128 issues operational commands, such asrotation or various speed commands, to the respective joint actuators(see FIGS. 3 and 4), in order to implement the behavior as instructed bythe behavior scheduling unit 127. As a result, the scheduled or expectedbehavior of the robot 100 may be implemented. The rotational positionsof the respective joint actuators are measured by the respective jointangle sensors 132 and used for feedback control.

The operation controller 128 may dynamically generate the operationalpattern for implementing sequentially commanded actions in real-time.Alternatively, it may calculate the trajectory schedules for walking orother main operational patterns off-line at the outset. In the lattercase, if a behavioral command is issued from the behavior schedulingunit 127, the operation controller 128 may call out the trajectoryschedule of the corresponding main operational patterns to implement thebehavior as it corrects the target trajectory as necessary. In theJapanese Laying-Open Patent Publication S-62-97006, there is disclosed amulti junction walking robot control apparatus which, by employingpre-stored walking pattern data, simplifies the control program anddensely joins the data of the walking pattern together.

The self-diagnosis unit 129 effects self-diagnosis of the inner statesof the robot 100, such as states of implementation of the scheduledbehavior, abnormalities, malfunctions or troubles, based on thedetection outputs of the sensors, such as the orientation sensor 130,temperature sensor 131, joint angle sensor 132, contact sensor 133,force sensor 134, power source management unit 135 and the communicationcontrolling detection unit 136. The results of the self-diagnosis can betransferred to the information recording management unit 125.

The information recording management unit 125 is able to acquire thefollowing items from various components:

(1) the user authentication information from the authenticationprocessor 112 and/or from the authentication processor 114;

(2) the contents of the dialog managed by the dialog management unit 124or the latest user command;

(3) the contents of the dialog or the user command and the behavioralschedule determined by the outer field or the environment; and

(4) the state of implementation of the behavioral schedule diagnosed bythe self-diagnosis unit 129.

The information recording management unit 125 combines these data andalso the current time furnished from a clock 126 for saving as the“behavioral history” or “empirical information”, that is as behaviorlog.

The behavior log is retained in a non-volatile fashion in a local memoryof the information recording management unit 125 or in a local disc, notshown. Alternatively, if the behavior log is stored in a memory devicethat can be inserted or removed like a cartridge, such as a memorystick, the behavior log may be dismounted from the robot 100 later foranalyzing and processing the log on an external computer system.

The “behavioral history” or the “empirical information” stored in theinformation recording management unit 125 may be retrieved using thetime information. Alternatively, the “behavioral history” or the“empirical information” may be acquired from the dialog with the robot100 through e.g., the speech recognition unit 115.

FIG. 6 shows another typical functional block structure for implementingthe recording processing of the behavioral records or empiricalinformation in the legged mobile robot 100. The robot 100, shown in FIG.6, is able to perform the operation not only of the “dialogue driving”system, in which the behavioral schedule is set in accordance with thedialog with the user based on the input speech or image, but also of the“autonomous driving” in which the behavioral schedule is setautonomously without recourse to the dialog with the user.

The legged mobile robot 100 utilizes at least one of the fingerprintsensor 111 and the visual sensor 113 for user authentication. As thevisual sensor 113, the image input device 251, such as a CCD (chargecoupled device) loaded on the head part may be used. Although not shownin FIGS. 1 to 4, a fingerprint sensor 111 includes a fingerprint readouthead on a site on the robot 100 on which the user can easily touch witha finger, such as head part 104 or a shoulder part.

The fingerprint sensor 111 and the authentication processor 112 are ableto perform more accurate authentication processing for the user whodefinitely requested fingerprint collation. On the other hand, thevisual sensor 113 and the authentication processor 114 are able toperform authentication processing, in the absence of the request on thepart of the user, with the user being not aware of it, as the user istracked and imaged with a camera. The authentication processors 112, 114are also able to transfer the results of authentication of the user toan information recording management unit 125.

In order for the user to input commands or data, the legged mobile robot100 of the present invention furnishes four means, namely a speechrecognition unit 115, a communication control unit 116, an imagerecognition unit 117 and a commander 118.

The speech recognition unit 115 is made up of a speech input unit 252,such as a microphone, a CPU 211 capable of recognizing and processingthe speech, or other calculation processing circuit. If the user inputsthe speech, the speech recognition unit 115 recognizes the user'sspeech. The command interpreting unit 119 also comprehends and analyzesthe command to the robot 100, based on the result of speech recognition.

The communication control unit 116 is made up of a communicationinterface 254, the CPU 211 issuing communication commands or processingdata or the like calculating circuit. The communication interface 254 isinterconnected over radio data communication network to exteriorinformation terminals, such as personal computers. The communicationcontrol unit 116 receives and processes the commands transmitted fromthe user over the communication network. The received command isconstrued in the command interpreting unit 119 and subsequentlytransmitted to the dialog management unit 124.

The image recognition unit 117 is made up of the image input device 251,such as a CCD camera, and a CPU 211, capable of image recognitionprocessing, or the like calculating circuit. The user is able to input acommand such as gesture or hand movements. The image recognition unit117 recognizes an image, while the command interpreting unit 119comprehends and analyzes the gesture or hand movements as a commandbased on the recognized result.

The commander 118 is constructed as a key/button operable user inputdevice, loaded on the legged mobile robot 100, or as a remotecontroller, not shown. The user can enter a command of a form that canbe uniquely interpreted by a robot, or a command of more abstruse form,on the commander 118. The user command, input on the commander 118, isinterpreted by the command interpreting unit 119 and thence sent to thedialog management unit 124.

The dialog management unit 124 chronologically manages the commands ofvariable forms, received through the command interpreting unit 119, tocomprehend the context of the dialog with the user, as well as tomaintain the dialog context. The dialog management unit 124 is able totransfer the contents of the dialog with the user to the informationrecording management unit 125. The dialog management unit 124 generatesa reply to the user input in a speech synthesis unit 120 andsubsequently outputs the reply to outside via a loudspeaker or outputsthe reply through a communication controller 121 to an external computersystem.

The speech synthesis unit 120 is made up of a speech outputting unit253, such as a loudspeaker, a CPU 211 that is able to synthesize thespeech, or the like calculation circuit. The communication controller121 is made up of a communication interface 254, a CPU 211, forprocessing commands or data exchanged over the communication interface254, and other calculating circuits.

An orientation sensor 130, a temperature sensor 131, a joint anglesensor 132, a contact sensor 133, a force sensor 134, a power sourcemanagement unit 135 and a communication controlling detection unit 136are functional modules for detecting the outside field or environment ofthe robot 100, changes in such outside field or environment and thestate of operation implementation.

The orientation sensor 130 detects the orientation of the robot 100 withrespect to the robot 100. The joint angle sensor 132 is equivalent tothe rotary encoder mounted in each joint actuator. The contact sensor133 is equivalent to ground touching confirming sensors 352, mounted onthe foot parts. The force sensor 134 is mounted in each part of thewhole body of the robot 100 for detecting the possible conflict againstthe user or obstacles of the outer field. The power source managementunit 135 is designed to monitor the power source voltage, current or thepower source capacity of the battery as a main power source of the robot100. The communication controlling detection unit 136 is made up of acommunication interface 254 and a CPU 211 or the like calculationcircuit for processing communication commands or data exchanged over thecommunication interface 254, and detects changes in the outer field orthe environment by the communication commands or communication data.

The environmental factors of the robot 100, defined by these sensors,are sent to an autonomous behavior scheduling unit 140 and to aself-diagnosis unit 129.

The autonomous behavior scheduling unit 140 determines a feeling modelin a feeling model status machine 141, based on changes in the externalenvironment, caused e.g., by input data from the above sensors, to set abehavioral schedule for the robot 100 autonomously to command theoperation controller 128 to implement the behavior. The autonomousbehavior scheduling unit 140 is also able to transfer the contents ofthe behavioral schedule to the information recording management unit125. Meanwhile, in the specification of the JP Patent ApplicationH-11-341374, already transferred to the present Assignee, there isdisclosed a legged robot having an operation-related feeling instinctmodel, which feeling instinct model is varied based on the inputinformation to control the operation.

For implementation of the behavior as commanded by the autonomousbehavior scheduling unit 140, the operation controller 128 issuesoperational commands, such as rotation command or various speedcommands, to the respective joint actuators. As a result, the scheduledor expected behavior of the robot 100 is implemented. The rotationalpositions of the respective joint actuators are measured by therespective joint angle sensors 132 and used for feedback control.

The operation controller 128 may dynamically generate operationalpatterns for implementing the sequentially commanded behaviors inreal-time. Alternatively, the trajectory schedule for main behavioralpatterns, such as walking, may be provided at the outset.

The self-diagnosis unit 129 effects self-diagnosis of the inner statesof the robot 100, such as states of implementation of the scheduledbehavior, abnormalities, malfunctions or troubles, based on thedetection outputs of the sensors, such as the orientation sensor 130,temperature sensor 131, joint angle sensor 132, contact sensor 133,force sensor 134, power source management unit 135 and the communicationcontrolling detection unit 136. The results of the self-diagnosis can betransferred to the information recording management unit 125.

The information recording management unit 125 is able to acquire thefollowing items from various components:

(1) the user authentication information from the authenticationprocessor 112 and/or from the authentication processor 114;

(2) the contents of the dialog managed by the dialog management unit 124or the latest user command;

(3) the behavioral schedule determined by the autonomous behaviorscheduling unit 140 in accordance with the feeling model; and

(4) the state of implementation of the behavioral schedule diagnosed bythe self-diagnosis unit 129.

The information recording management unit 125 combines these data andalso the current time furnished from a clock 126 for saving as the“history of behavior” or “empirical information”, that is as behaviorlog.

The behavior log is retained in a non-volatile fashion in a local memoryof the information recording management unit 125 or in a local disc, notshown. Alternatively, if the behavior log is stored in a memory devicethat can be inserted or removed like a cartridge, such as a memorystick, the behavior log may be dismounted from the robot 100 later foranalyzing and processing the log on an external computer system.

If both the “dialogue driving” system, in which the behavioral schedulefor the robot 100 is set in accordance with the dialog with the userbased on the speech or image input, and the “autonomous driving” system,in which the behavioral schedule is set autonomously without recourse tothe dialog with the user, are to be supported, the discriminationinformation indicating whether the behavioral schedule is of thedialogue driving type or the autonomous driving type may be recordedtogether.

The “behavioral history” or the “empirical information” stored in theinformation recording management unit 125 can be retrieved using thetime information. Alternatively, it can be derived from the dialog withthe robot 100 through e.g., the speech recognition unit 115.

FIG. 7 shows a data format of the behavior log. As shown therein, eachlog is made up of a time field, a user authentication information field,a dialog contents field, a behavioral schedule field, an action modefield, and a result of behavior—result of self-diagnosis field.

The actual time supplied from the clock 126 is written in the timefield. In the user authentication information field is written the userinformation, authenticated by the fingerprint sensor 111 andauthentication processor 112 or by the visual sensor 113 and theauthentication processor 114. The authenticated user may be anoriginator of the command on the dialog process. In the dialog contentsfield are written the dialog contents supervised by the dialogmanagement unit 124.

In the behavior schedule filed is written the behavioral schedule as setby the behavior scheduling unit 127 or the autonomous behaviorscheduling unit 140. In the action mode field is written the informationfor discriminating with which of the dialog driving system and theautonomous driving system the current behavioral schedule of the robot100 has been set.

In the result of behavior—result of self-diagnosis field is written thestate of operation implementation, as calculated based on the output ofeach joint angle sensor 132 or the result of self-diagnosis, asdetermined based on outputs of other sensors, such as malfunction state.

It should be noted that the analysis of the behavior log inclusive ofthe above data leads to estimation of the behavior of the robot 100 andcauses of abnormalities, accidents or troubles.

FIG. 8 shows the state of actions that can be taken by the robot 100 ofthe present invention. As shown therein, the robot 100 of the presentinvention can take the states of active driving and passive driving.

In the passive driving, the robot receives commands, interpretedunequivocally, from the user, and implements the operation only inaccordance with the commands, while the inner states, such as thinkingor feeling, proper to the robot 100, are not active.

For example, such driving state in which a pre-stored walking pattern isread out responsive to the user's command “walk” to implement thewalking movement corresponds to the passive driving. In this case,although the control function of the robot 100 performs correction ofthe target trajectory accompanying stable orientation control ordisturbances, the thinking or feeling is not in operation, such that thedialog with the user is not valid or is not needed.

Thus, in the passive driving state, the input command is directlywritten in the dialog contents field of the behavior log shown in FIG.7. Since the robot 100 itself does not set a behavioral schedule, blankor default values are written in the behavioral schedule field. In theresult of behavior—result of self-diagnosis field is written the stateof operation implementation, as calculated based on the output of eachjoint angle sensor 132, or the result of self-diagnosis, as determinedbased on outputs of other sensors, such as malfunction state.

In the active state, there are two operating states, namely the “dialogdriving” system of setting the behavioral schedule in compliance withthe dialog with the user, based on an audio or image input, and the“autonomous driving” system of autonomously setting the behavioralschedule without recourse to the dialog with the user.

The robot 100 may operate in accordance with one of the dialog drivingsystem and the autonomous driving system. Alternatively, the two drivingsystems may be changed over unidirectionally or bidirectionally. Forexample, transition may be made to the autonomous driving system subjectto occurrence of a preset event, such as time elapsed as from the lastuser input or the from the last dialog, or reversion may be made to thedialog driving system subject to the next user input or to theoccurrence of the next dialog.

FIG. 9 schematically shows the operational sequence of the robot 100during active driving.

If, during dialog driving, a user input is made, this is recognized andinterpreted as a command for dialog control. The user who makes thedialog is authenticated through a fingerprint sensor or visual sensor. Abehavioral schedule is set based on the contents of the dialog and theoutside environment, input from the sensor, to implement the behavioralpattern corresponding to the behavior. The state of implementation ofthe behavioral pattern is obtained by a sensor input.

Thus, in the dialog driving state, the authentication informationconcerning the user making a dialog with the robot 100 is written in theuser authentication information field in the behavior log shown in FIG.7. In the behavior contents field are written the contents of the dialogestablished between the robot and the user. In the behavioral schedulefield, the behavioral schedule as set for the robot 100 depending on thecontents of the dialog or responsive to changes in the outsideenvironment. In the operation mode field, the discrimination informationtestifying to the dialog driving state is written in the operationalmode field. In the result of behavior—result of self-diagnosis field iswritten the state of operation implementation, as calculated based onthe output of each joint angle sensor 132, or the result ofself-diagnosis, as determined based on outputs of other sensors, such asmalfunction state.

In the autonomous driving, the user input is interrupted, or the userinput is not accepted, so that the dialog with the user is not valid. Insuch case, the robot 100 autonomously sets the behavioral schedule,based on changes in the outside environment, input from the sensor, andon the feeling model, without recourse to the dialog with the user. Theoperational pattern corresponding to the behavior is implemented, andthe state of implementation of the operational pattern is obtained basedon the sensor input. During the autonomous driving, the near-by user isnot the command originator, so that user authentication may or may notbe performed.

Thus, during the autonomous driving state, the authenticationinformation of the near-by user is written in the user authenticationinformation field, or the blank or default value may be in the field. Inthe dialog contents field is directly written the input command. In thebehavioral schedule is written the behavioral schedule determined by therobot 100 responsive to the feeling model or to changes in the outsideenvironment. In the operation mode field is written the discriminationinformation testifying to the dialog driving state. In the result ofbehavior—result of self-diagnosis field is written the state ofoperation implementation, as calculated based on the output of eachjoint angle sensor 132, or the result of self-diagnosis, as determinedbased on outputs of other sensors, such as malfunction state.

It should be noted that the analysis of the behavior log inclusive ofthe above data leads to estimation of the behavior of the robot 100 andcauses of abnormalities, accidents or troubles, regardless of which ofthe driving states shown in FIG. 8 is assumed by the robot 100.

Although the present invention has been elucidated with reference toseveral preferred embodiments, it is to be noted that those skilled inthe art can modify the embodiments or use technical equivalents withoutdeparting from the scope of the invention.

The foregoing des is merely illustrative and should not be construed ina limiting fashion. The purport of the present invention can be bestunderstood by having reference to the claims.

INDUSTRIAL APPLICABILITY

According to the present invention, as described above, there may beprovided a behavioral schedule setting type robot of superior qualitywhich is able to set a behavioral schedule in compliance with the dialogwith the user based on an audio or image input or to autonomously set abehavioral schedule without recourse to a user input.

There may also be provided a behavioral schedule setting type robot ofsuperior quality which is able to search into a cause of abnormalities,malfunctions or troubles in which the robot is involved during “dialogdriving” or “autonomous driving”.

The robot of the present invention is able to authenticate a user whoissues a command to the robot. The robot of the present invention alsoextracts the biological feature information, such as the face or voiceof the user who cannot be authenticated, or records the contents of thecommand issued from the user in combination with the behavior taken bythe robot responsive to the command and the time of implementation ofthe behavior. The inner state of the robot as well as the sensor inputinformation can also be recorded in combination. As a result, the useris able to search into the cause of the abnormalities, malfunctions ortroubles that occurred in the robot by analyzing the recorded contents.

What is claimed is:
 1. A controlling apparatus for a robot of a typeformed by a plurality of joint actuators and operating in accordancewith a behavioral schedule, comprising: a behavior scheduling unit forsetting the behavioral schedule of the robot; an operation controllerfor implementing an operational pattern corresponding to the behavioralschedule determined by said behavior scheduling unit by driving each ofthe joint actuator; a detector for detecting a state of operationimplementation by said operation controller; and a recording unit forrecording a log including the behavioral schedule by said behaviorscheduling unit and the state of operation implementation by saiddetector.
 2. The controlling apparatus for the robot according to claim1 further comprising: a user input unit for receiving a command or datafrom a user, and a dialog management unit for supervising a dialog withthe user based on user input command or data from said user input unit,wherein said behavior scheduling unit sets the behavioral schedule inaccordance with contents of the dialog in said dialog management unit,and said recording unit takes a log of the dialog contents.
 3. Thecontrolling apparatus for the robot according to claim 1 furthercomprising a self-diagnosis unit for diagnosing each part of the robot,wherein said recording unit takes the log of results of aself-diagnosis.
 4. The controlling apparatus for the robot according toclaim 1 further comprising a user authentication unit for authenticatinga user lying in a vicinity of the robot, wherein said recording unittakes a log of results of user authentication.
 5. The controllingapparatus for the robot according to claim 1, wherein said behaviorscheduling unit sets the behavioral schedule based on a feeling model asdetermined in response to external changes.
 6. The controlling apparatusfor the robot according to claim 1, wherein said behavior schedulingunit is operable in accordance with a first operating system of settingthe behavioral schedule based on the dialog with the user or inaccordance with a second operating system of setting the behavioralschedule based on the feeling model as determined in response to theexternal changes, and said recording unit takes a log of the operatingsystems in said behavior scheduling unit.
 7. A controlling method for arobot of a type formed by a plurality of joint actuators and operatingin accordance with a behavioral schedule, comprising: a behaviorscheduling step of setting the behavioral schedule of the robot; anoperation controlling step of implementing an operational patterncorresponding to the behavioral schedule determined by said behaviorscheduling step by driving each joint actuator; a detecting step ofdetecting a state of operation implementation by said operationcontrolling step; and a recording step of recording a log including thebehavioral schedule by said behavior scheduling step and the state ofoperation implementation detected by said detecting step.
 8. Thecontrolling method for the robot according to claim 7 furthercomprising: a user input step of receiving a command or data from auser, and a dialog management step of supervising a dialog with the userbased on the user input command or data from said user input step,wherein said behavior scheduling step comprises a step of setting thebehavioral schedule in accordance with contents of the dialog in saiddialog management step, and said recording step comprise a step oftaking a log of the dialog contents.
 9. The controlling method for therobot according to claim 7 further comprising a self-diagnosis step ofdiagnosing each part of the robot, wherein said recording step comprisea step of taking the log of results of the self-diagnosis.
 10. Thecontrolling method for the robot according to claim 7 further comprisinga user authentication step of authenticating a user lying in a vicinityof the robot, wherein said recording step comprise a step of taking alog of the results of user authentication.
 11. The controlling methodfor the robot according to claim 7 wherein said behavior scheduling stepcomprise a step of setting the behavioral schedule based on a feelingmodel determined in response to external changes.
 12. The controllingmethod for the robot according to claim 7 wherein said behaviorscheduling step is operable in accordance with a first operating systemof setting the behavioral schedule based on the dialog with the user orin accordance with a second operating system of setting the behavioralschedule based on the feeling model determined in response to theexternal changes, and said recording step comprises a step of taking alog of the operating systems in said behavior scheduling step.